Data Fingerprinting -- Identifying Files and Tables with Hashing Schemes

Master's thesis in Computer scienceINTRODUCTION: Although hash functions are nothing new, these are not limited to cryptographic purposes. One important field is data fingerprinting. Here, the purpose is to generate a digest which serves as a fingerprint (or a license plate) that uniquely identifies a file. More recently, fuzzy fingerprinting schemes — which will scrap the avalanche effect in favour of detecting local changes — has hit the spotlight. The main purpose of this project is to find ways to classify text tables, and discover where potential changes or inconsitencies have happened. METHODS: Large parts of this report can be considered applied discrete mathematics — and finite fields and combinatorics have played an important part. Rabin’s fingerprinting scheme was tested extensively and compared against existing cryptographic algorithms, CRC and FNV. Moreover, a self-designed fuzzy hashing algorithm with the preliminary name No-Frills Hash has been created and tested against Nilsimsa and Spamsum. NFHash is based on Mersenne primes, and uses a sliding window to create a fuzzy hash. Futhermore, the usefullness of lookup tables (with partial seeds) were also explored. The fuzzy hashing algorithm has also been combined with a k-NN classifier to get an overview over it’s ability to classify files. In addition to NFHash, Bloom filters combined with Merkle Trees have been the most important part of this report. This combination will allow a user to see where a change was made, despite the fact that hash functions are one-way. Large parts of this project has dealt with the study of other open-source libraries and applications, such as Cassandra and SSDeep — as well as how bitcoins work. Optimizations have played a crucial role as well; different approaches to a problem might lead to the same solution, but resource consumption can be very different. RESULTS: The results have shown that the Merkle Tree-based approach can track changes to a table very quickly and efficiently, due to it being conservative when it comes to CPU resources. Moreover, the self-designed algorithm NFHash also does well in terms of file classification when it is coupled with a k-NN classifyer. CONCLUSION: Hash functions refers to a very diverse set of algorithms, and not just algorithms that serve a limited purpose. Fuzzy Fingerprinting Schemes can still be considered to be at their infant stage, but a lot has still happened the last ten years. This project has introduced two new ways to create and compare hashes that can be compared to similar, yet not necessarily identical files — or to detect if (and to what extent) a file was changed. Note that the algorithms presented here should be considered prototypes, and still might need some large scale testing to sort out potential flaw

Evaluation of factors influencing disinfection efficacy for aquaculture

Prosjektleder Aina Charlotte WennbergDisinfectants falls within the scope of the Biocidal Products Regulation (BPR), however, there are no specific guidance on how to test the efficacy of disinfectants for use in aquaculture. The Norwegian Medicines Agency and the Norwegian Food Safety Authority are the current authorities for approval and follow up on the national regulations of disinfectants for aquaculture in Norway, while the Norwegian Environmental Agency (NEA) will be the authority for the BPR. Thus, this project was funded by NEA for making a guidance document for efficacy testing of disinfectants for use in aquaculture that is in accordance with the BPR and that can be a supplement to the European Chemical Agency’s (ECHA) guidance on the Biocidal Products Regulation Volume II Efficacy assessment and evaluation (Part B + C). This report summarizes the aquaculture industry in EU and Norway, and the typical conditions for uses and use patterns for disinfectants in aquaculture. This information was used to make a recommended guidance document on how to perform efficacy testing of disinfectants for use in the EU and EEA in accordance with BPR. The guidance document is a standalone document.Norwegian Environment AgencypublishedVersio

Comparing flow cytometry and microscopy in the quantification of vital aquatic organisms in ballast water, Journal of Marine Engineering & Technology

The ability to quantify vital aquatic organisms in the 2–50 µm size range was compared between five different flow cytometers and several different microscopes. Counts of calibration beads, algal monocultures of different sizes as well as organisms in a Wadden Sea sample were compared. Flow cytometers and microscopes delivered different bead concentrations. These differences between the instruments became larger for algal monocultures and were even higher for the Wadden Sea sample. It was observed that the concentration differences were significant between flow cytometer and microscope counts, and that this difference increased with the size of the objects counted. Microscope counts were more accurate for larger (50 µm) objects because cytometers struggled with bigger particles that clogged the instruments. Contrary to microscopy, the flow cytometers were capable of accurately enumerating cultured cells in the 2–10 µm size range and cells in the lower size range of the 10–50 µm size class. Flow cytometers were also well-suited to assess low abundance samples due to their ability to process larger volumes than microscopes. The results were used to indicate which tools are suitable for ballast water monitoring: flow cytometry is a suitable technology for an indicative and real time analysis of ballast water samples whilst only microscopy would be robust enough for detailed taxonomical analyses.publishedVersio

Comparing flow cytometry and microscopy in the quantification of vital aquatic organisms in ballast water

The ability to quantify vital aquatic organisms in the 2–50 µm size range was compared between five different flow cytometers and several different microscopes. Counts of calibration beads, algal monocultures of different sizes as well as organisms in a Wadden Sea sample were compared. Flow cytometers and microscopes delivered different bead concentrations. These differences between the instruments became larger for algal monocultures and were even higher for the Wadden Sea sample. It was observed that the concentration differences were significant between flow cytometer and microscope counts, and that this difference increased with the size of the objects counted. Microscope counts were more accurate for larger (50 µm) objects because cytometers struggled with bigger particles that clogged the instruments. Contrary to microscopy, the flow cytometers were capable of accurately enumerating cultured cells in the 2–10 µm size range and cells in the lower size range of the 10–50 µm size class. Flow cytometers were also well-suited to assess low abundance samples due to their ability to process larger volumes than microscopes. The results were used to indicate which tools are suitable for ballast water monitoring: flow cytometry is a suitable technology for an indicative and real time analysis of ballast water samples whilst only microscopy would be robust enough for detailed taxonomical analyses

Comparing flow cytometry and microscopy in the quantification of vital aquatic organisms in ballast water

The ability to quantify vital aquatic organisms in the 2–50 µm size range was compared between five different flow cytometers and several different microscopes. Counts of calibration beads, algal monocultures of different sizes as well as organisms in a Wadden Sea sample were compared. Flow cytometers and microscopes delivered different bead concentrations. These differences between the instruments became larger for algal monocultures and were even higher for the Wadden Sea sample. It was observed that the concentration differences were significant between flow cytometer and microscope counts, and that this difference increased with the size of the objects counted. Microscope counts were more accurate for larger (50 µm) objects because cytometers struggled with bigger particles that clogged the instruments. Contrary to microscopy, the flow cytometers were capable of accurately enumerating cultured cells in the 2–10 µm size range and cells in the lower size range of the 10–50 µm size class. Flow cytometers were also well-suited to assess low abundance samples due to their ability to process larger volumes than microscopes. The results were used to indicate which tools are suitable for ballast water monitoring: flow cytometry is a suitable technology for an indicative and real time analysis of ballast water samples whilst only microscopy would be robust enough for detailed taxonomical analyses

Comparing flow cytometry and microscopy in the quantification of vital aquatic organisms in ballast water, Journal of Marine Engineering & Technology

The ability to quantify vital aquatic organisms in the 2–50 µm size range was compared between five different flow cytometers and several different microscopes. Counts of calibration beads, algal monocultures of different sizes as well as organisms in a Wadden Sea sample were compared. Flow cytometers and microscopes delivered different bead concentrations. These differences between the instruments became larger for algal monocultures and were even higher for the Wadden Sea sample. It was observed that the concentration differences were significant between flow cytometer and microscope counts, and that this difference increased with the size of the objects counted. Microscope counts were more accurate for larger (50 µm) objects because cytometers struggled with bigger particles that clogged the instruments. Contrary to microscopy, the flow cytometers were capable of accurately enumerating cultured cells in the 2–10 µm size range and cells in the lower size range of the 10–50 µm size class. Flow cytometers were also well-suited to assess low abundance samples due to their ability to process larger volumes than microscopes. The results were used to indicate which tools are suitable for ballast water monitoring: flow cytometry is a suitable technology for an indicative and real time analysis of ballast water samples whilst only microscopy would be robust enough for detailed taxonomical analyses